Story Cloze Evaluator: Vector Space Representation Evaluation by Predicting What Happens Next

The main intrinsic evaluation for vector space representation has been focused on textual similarity, where the task is to predict how semantically similar two words or sentences are. We propose a novel framework, Story Cloze Evaluator, for evaluating vector representations which goes beyond textual similarity and captures the notion of predicting what should happen next given a context. This evaluation methodology is simple to run, scalable, reproducible by the community, non-subjective, 100% agreeable by human, and challenging to the state-of-theart models, which makes it a promising new framework for further investment of the representation learning community.